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PREFACE 


This  research  was  funded  and  supported  by  the  Air  Force  Research  Laboratory  at  Wright- 
Patterson  Air  Force  Base  and  carried  out  at  the  Man  Vehicles  Laboratory  at  the 
Massachusetts  Institute  of  Technology.  The  study  was  conducted  under  the  supervision  of 
Dr.  Laurence  Young  of  the  Apollo  Program  of  Astronautics  at  MIT.  Data  analysis  support 
was  provided  by  Dr.  Alan  Natapoff  of  the  Department  of  Aeronautics  and  Astronautics  at 
MIT.  Additional  software  and  technical  support  was  supplied  by  Mr.  Robert  Esken  and  the 
Human  Effectiveness  Directorate  of  AFRL. 


INTRODUCTION 


Spatial  disorientation  and  loss  of  situational  awareness  continue  to  be  a  leading  cause  of 
military  aircraft  accidents.  Currently,  pilots  are  dependent  solely  on  their  visual  and 
vestibular  systems  for  attitude  information,  both  of  which  can  be  easily  misled.  The 
proposed  solution  to  this  spatial  disorientation  problem  is  to  increase  the  avenues  through 
which  information  is  given  to  the  pilot.  This  can  be  done  through  the  development  of  the 
helmet-mounted  display  (HMD)  to  provide  better  and  continuous  visual  cues,  the  use  of 
three-dimensional  audio  for  target  detection  and  terrain  collision  avoidance,  and  the  use  of 
tactile  sensors  for  attitude  information.  This  study  will  look  at  the  HMD  and  the  use  of  3-D 
audio  for  target  detection. 

HMDs  allow  the  pilot  to  look  longer  off-boresight  during  air-to-air  and  air-to-ground  tasks. 
Research  has  been  conducted  to  develop  HMD  symbology  that  provides  effective  target 
cueing  information  while  minimizing  visual  occlusion.  The  Non-Distributed  Flight 
Reference  (NDFR)  symbology  has  proven  to  be  an  effective  symbology  in  allowing  the  pilot 
to  look  longer  off-boresight  while  maintaining  flight  performance  on  par  with  current 
military  standard  displays.  Research  into  auditory  localization  has  shown  that  target 
acquisition  time  decreased  significantly  with  the  use  of  a  combined  3-D  audio  and  visual 
cueing  system.  The  objective  of  this  study  is  to  determine  the  combined  effects  of  the  two 
systems  by,  first,  looking  at  the  flight  performance  benefits  of  the  NDFR  symbology;  and 
second,  by  studying  the  target  acquisition  benefits  of  using  a  combined  3-D  audio/ visual 
cueing  system  during  a  visual  target  search  task. 

Background 

Past  research  has  shown  that  spatial  orientation  and  situational  awareness  benefits  are  related 
to  the  use  of  HMD  symbology  and  increased  performance  with  localized  audio.  The 
research  on  HMDs  includes  looking  at  the  benefits  of  off-boresight  symbology  for  target 
search  and  attack  tasks,  developing  a  methodology  for  evaluating  off-axis  HMD  ownship 
information,  and  the  development  and  evaluation  of  the  NDFR  symbology  for  on/ off- 
boresight  viewing.  The  localized  auditory  research  includes  looking  at  the  benefits  of 
augmenting  the  Terrain  Collision  Avoidance  System  (TCAS)  with  3-D  audio  cues,  using 
audio  cues  both  spoken  and  non-spoken  to  help  guide  a  visual  search,  and  the  effects  of  high 
acceleration  on  audio  localization.  The  conclusions  from  this  research  are  described  below. 
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Helmet-Mounted  Display  Research 


The  Air  Force  currently  equips  most  combat 
aircraft  with  a  Head-Up  Display  (HUD)  which 
gives  the  pilot  attitude  information  while 
looking  on  boresight  (straight  ahead).  The 
HUD  however  fails  to  give  the  pilot  any 
attitude  information  during  tasks  that  require 
them  to  look  off-boresight.  Therefore  the  use 
of  HMDs  was  proposed  as  a  means  to 
continuously  provide  the  pilot  with  attitude 
information.  Nonetheless  having  continuous 
attitude  information  through  an  HMD  does 
not  equate  to  better  performance  or  an  increase 
in  situational  awareness.  Figure  1:  Military  Standard  HUD 


For  this  reason  the  Air  Force  conducted  studies  comparing  the  use  of  HMDs  to  the  current 
HUD.  In  one  such  study,  Geiselman  and  Osgood1  compared  the  utility  of  three  off- 
boresight  helmet-mounted  display  symbology  information  levels  for  high  angle  target  search 
and  intercept  during  a  simulated  air-to-air  engagement.  The  information  levels  included:  (1) 
HUD  presentation  of  both  ownship  status  and  target  location;  (2)  HUD  status  plus  HMD 
target  location;  and  (3)  HUD  status  plus  HMD  target  location  plus  HMD  ownship  status. 
Four  different  attitude  symbology  elements  were  evaluated  within  ownship  status  level.  The 
experiment  contained  two  phases.  The  first  was  a  target  search  task  where  the  subjects 
visually  searched  the  surrounding  airspace  in  order  to  locate  and  acquire  radar  lock  on  an 
airborne  target  while  maintaining  their  initial  aircraft  parameters  of  5000  feet  altitude,  480 
knots  airspeed,  and  heading  of  0°.  The  second  phase  was  the  attack  phase  where  the 
subjects  maneuvered  the  aircraft  to  bring  the  target  within  the  launch  envelope;  the  trial  was 
successful  when  a  good  launch  was  accomplished.  During  both  the  search  and  attack  phases, 
pilots  were  able  to  look  longer  off-boresight  with  the  use  of  the  HMD  without  any  decrease 
in  performance.  “The  capability  and  comfort  to  search  more  surrounding  area  for  a  longer 
period  of  time,  while  not  accruing  a  performance  cost,  may  be  interpreted  as  a  potential 
[situational  awareness]  benefit.”1  A  modification  of  the  Subjective  Workload  Dominance 
Technique  (SWORD)  was  used  to  record  subjective  ratings  of  situation  awareness  benefits. 
The  SWORD  data  found  no  differences  among  symbology  formats  within  the  HMD 
ownship  status  level.  However,  subjects  stated  that  ownship  status  information  should  be 
included  in  the  symbology. 

The  previous  study  showed  that  pilots  with  HMD  spent  more  time  off-boresight  versus 
pilots  using  a  HUD;  hence  the  HMD  should  display  information  that  pilots  would  normally 
obtain  from  looking  forward  into  the  cockpit.  This  information  is  intended  to  keep  the  pilot 
spatially  oriented.  A  study  conducted  by  Geiselman,  Havig,  and  Brewer2  describes  the 
design  and  evaluation  of  the  NDFR  symbology,  which  provides  the  pilot  with  continuous 
ownship  status  information,  for  off-boresight  viewing.  Three  symbology  sets  were 
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compared:  the  standard  HUD  symbology,  the  Visually  Coupled  Acquisition  Targeting 
System  (V CATS),  and  the  NDFR  symbology. 


Figure  2:  Military  Standard  HUD,  VCAT,  NDFR 

The  experiment  consisted  of  the  subjects  recalling  ownship  information  from  static 
presentations  of  the  symbology  sets  for  specified  exposure  times.  The  subjects  also  gave 
subjective  feedback  concerning  the  sets,  such  as  their  confidence  in  determining  ownship 
status.  The  results  show  that  the  NDFR  symbology  afforded  better  information 
interpretation.  The  NDFR  appears  to  provide  extra  information  processing  capacity 
compared  to  the  VCATS  symbology.  Subjective  feedback  is  consistent  with  these  findings 
in  that  subjects  felt  more  confident  in  determining  ownship  information  with  the  NDFR 
display.  The  ability  for  pilots  to  quickly  assess  ownship  information  to  establish  an  accurate 
perception  of  orientation  may  be  a  possible  situational  awareness  benefit.  Use  of  the  NDFR 
symbology  appears  to  reduce  pilot  workload  when  establishing  spatial  orientation,  allowing 
the  pilot  to  perform  other  tasks  while  maintaining  flight  performance. 

Further  studies  by  Havig,  Jenkins,  and  Geiselman  looked  at  how  attitude  information  should 
be  displayed  when  off-boresight;  two  options  are  forward  or  line  of  sight  (LOS).3  The 
“forward”  symbology  displays  on-boresight  attitude  information  when  looking  off-boresight, 
while  the  “LOS”  symbology  displays  attitude  information  congruent  with  the  visual  scene. 
They  investigated  five  different  symbologies  (standard  HUD,  VCATS,  ASAR,  Theta  Ball, 
and  NDFR).  The  experiment  consisted  of  two  different  tasks,  with  the  pilots  performing  the 
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task  either  facing  the  monitor  or  rotated  90°  and  looking  over  their  shoulder  (off-bore sight). 
In  the  first  task,  pilots  maintained  straight  and  level  flight  with  simulated  turbulence.  The 
second  task  had  pilots  interpret  a  static  representation  of  their  attitude  and  respond  via  a  key 
press;  then  the  display  went  live  and  they  had  to  fly  to  a  new  commanded  attitude.  The 
NDFR  symbology  resulted  in  better  control  when  off-boresight  and  forward  was  better  than 
LOS.  The  reaction  times  for  choosing  the  correct  attitude  in  the  second  part  of  the 
experiment  were  fastest  using  the  NDFR.  The  NDFR  during  attitude  recovery  provided  a 
significantly  faster  initial  stick  input  even  though  the  overall  performance  was  equal  for  all 
symbol  sets.  The  NDFR  symbology  performed  as  well  if  not  better  in  providing  ownship 
information.  Pilots  were  able  to  keep  a  more  accurate  perception  of  orientation  when  using 
the  NDFR  symbology.  Also,  the  faster  reaction  times  may  allow  pilots  to  recover  from 
situations  where  they  may  be  spatially  disoriented. 

The  NDFR  symbology  allows  pilots  to  look  longer  off-boresight  during  air-to-air  and  air-to- 
ground  tasks  during  simulated  trials.  A  study  by  Jenkins,  Thurling,  Havig,  and  Geiselman4 
looked  at  quantifying  pilot  performance  during  in-flight  operationally  representative  tasks:  air- 
to-ground,  air-to-air,  and  unusual  attitude  recoveries.  The  symbology  sets  evaluated  were  the 
standard  HUD,  the  NDFR,  and  the  VCATS.  The  China  Lake  Situational  Awareness  (CLSA) 
rating  scale  was  used  to  measure  pilot  situational  awareness  for  each  symbology  and  the 
Modified  Cooper-Harper  Rating  (MCHR)  scale  was  used  to  measure  workload.  Pilots  were 
able  to  spend  longer  time  looking  off-boresight  with  the  HMDs;  however,  the  NDFR  was 
the  only  one  that  had  no  decrease  in  performance.  This  was  true  for  both  air-to-ground  and 
air-to-air  tasks.  The  VCATS  symbology  allowed  the  pilots  to  look  longer  off-boresight,  but 
it  failed  to  provide  adequate  situational  awareness  resulting  in  a  decrease  in  performance.  For 
the  air-to-ground  task  the  situational  awareness  rating  was  equal  for  the  HUD  versus  the 
NDFR  symbology  sets  while  the  VCATS  performed  significantly  worse.  These  results  were 
also  seen  in  the  workload  assessment.  For  the  air-to-air  tasks  NDFR  had  the  best  situational 
awareness  and  workload  ratings.  Both  the  NDFR  and  the  VCATS  symbology  sets  allowed 
significantly  faster  reaction  times  for  first  significant  stick  input.  The  results  of  the  flight  test 
are  consistent  with  the  ground  simulation  trials  from  previous  studies. 
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Based  on  the  recommendations  of  the  flight  test,  changes  were  made  to  the  NDFR 
symbology.  Two  variants  of  the  NDFR  format  as  well  as  the  standard  FIUD  and  baseline 
NDFR  format  were  examined  during  two  simulated  operationally  representative  air-to-air 
intercept  tasks  that  employed  the  use  of  an  HMD  for  the  off-boresight  visual  acquisition  of  a 
target  aircraft.5  The  objective  of  the  study  was  to  evaluate  the  display  formats  for  off- 
boresight  HMD  use  in  conveying  rate  of  change  or  trend  information  to  the  pilot.  It  was 

hypothesized  that  pilots  would  maintain  better 
control  if  they  had  the  ability  to  detect  changes  in 
airspeed  and  altitude.  After  completing  both  trials,  a 
low-slow  intercept  and  a  High  Value  Airborne 
Assets  defense  trial,  the  pilots  were  asked  to  rate 
their  situational  awareness  using  the  CLSA  scale  and 
workload  using  the  MCHR  scale.  The 
NDFR/ Odometer  symbology  proved  to  be  the  best 
symbology  for  providing  trend  information  and 
received  the  best  MCHR  rating.  The 
NDFR/ Odometer  symbology  allowed  the  greatest 
time  off-boresight,  performed  equally  to  the  HUD 
and  other  NDFR  symbologies,  and  was  the  preferred  symbology  of  pilots. 

Geiselman  conducted  another  study  whose  purpose  was  to  develop  a  methodology  for 
evaluating  off-axis  HMD  ownship  information.6  Three  experiments  were  discussed  in  the 
paper;  one  (Geiselman,  Osgood,  1994)1  has  already  been  discussed.  The  other  two 
experiments  were  a  low  altitude  flight  task,  with  and  without  manipulation  of  off-axis 
ownship  information,  and  an  air-to-air  target  cueing  study. 

The  first  study  included  a  simulated,  low-level,  high-speed,  airborne 

surveillance/ reconnaissance  mission.  A  with  and  without  manipulation  of  a  simple  off-axis 
ownship  information  display  was  performed.  Pilots  were  instructed  to  maintain  a  400  ft,  480 
knot  flight  profile  along  a  prescribed  heading.  The  consequences  for  excess  altitude 
deviations  were  ground  collision  and  the  threat  of  surface-to-air  missiles.  Subjects 
maintained  altitude  and  heading  while  searching  for  airborne  threats  and  took  evasive  action 
if  fired  upon.  The  trial  consisted  of  a  search  phase  and  a  monitor  phase.  During  both  phases, 
HMD-presented  ownship  information  resulted  in  the  pilots  looking  farther  off-axis  for  a 
longer  period  of  time  without  any  decrease  in  performance.  Two  other  interesting  effects 
were  found.  During  the  142  trials  of  the  experiment,  no  ground  strikes  occurred  when 
HMD  ownship  information  was  available.  Without  the  HMD,  five  ground  strikes  were 
recorded.  Secondly,  a  possible  benefit  to  situational  awareness  was  seen  by  looking  at  a 
snapshot  of  the  pilot  behavior  at  the  instant  a  significant  event  occurred,  for  example  a 
hostile  missile  launch.  As  seen  in  Figure  4,  pilot  LOS  when  HMD  aided  tended  to  be  at 
angles  much  closer  to  the  location  of  the  threat  when  the  missile  was  launched.  In  fact, 
during  trials  where  HMD  information  was  not  available,  the  average  LOS  angle  actually 
resulted  in  the  hostile  aircraft  location  being  beyond  the  reasonable  field  of  view  of  the  pilot 
(left  graphic  in  Figure  4). 


Figure  3:  NDFR/Odometer 
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No  Helmet  Display  Symbology _ With  Helmet  Display  Symbology 


Figure  4:  Pilot  LOS  during  Critical  Event  with  and  without  an  HMD  Aid 

In  developing  the  methodology  for  evaluating  off-axis  HMD  ownship  information, 
Geiselman  et  al..  also  reviewed  another  experiment  which  investigated  the  effects  of  HMD- 
resident  target  location  information  reference  frame  during  air-to-air  target  acquisition  and 
intercept  tasks.  The  objective  of  the  study  was  to  determine  if  off-boresight  locator 
information  should  indicate  the  position  of  the  target  relative  to  the  nose  of  the  aircraft  (fly- 
to),  relative  to  the  nose  on  the  pilot’s  face  (look- to),  or  a  combination  of  the  two.  The  HMD 
did  include  ownship  status  information.  The  experimental  trials  consisted  of  a  search  phase 
and  an  attack  phase.  For  both  tasks  the  pilots  looked  farther  off-boresight  when  HMD 
target  location  information  was  available.  Although  not  supported  by  the  performance  data, 
the  subjective  results  strongly  suggested  that  the  pilots  favored  the  combination  (multiple 
coordinate  reference  frame). 

The  overall  conclusions  from  the  HMD  literature  can  be  summarized  as  follows:  1)  Pilots 
look  longer  off-boresight  when  HMD  is  available  compared  to  using  only  the  HUD  for  both 
air-to-air  and  air-to-ground  tasks.  2)  There  was  no  decrease  in  performance  due  to  looking 
longer  off-boresight.  3)  The  pilots  preferred  that  off-axis  ownship  status  information  be 
included  within  the  HMD  symbology  set.  4)  The  HMD  symbology  should  minimize  visual 
occlusion.  5)  The  NDFR  symbol  set  is  the  preferred  symbol  set  for  conveying  ownship 
information  in  the  HMD,  as  compared  to  the  standard  HUD  and  VCATS.  6)  When  looking 
off-boresight  the  symbology  should  be  forward  referenced.  7)  The  NDFR  should  include 
trend  information. 
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Localized  Audio  Research 


Pilots  are  saturated  with  visual  information.  However,  humans  have  at  their  disposal  two 
other  modalities  rarely  utilized,  auditory  and  tactile.  Auditory  cues  are  currently  used  in 
aircraft  primarily  as  warnings  and  alarms.  Localized  audio  is  being  developed  to  aid  in  target 
acquisition  and  detection,  improve  terrain  collision  avoidance,  and  help  with  traffic 
awareness. 

When  humans  hear  a  spatial  audio  sound  the  signal  is  evaluated  depending  on  its  elevation 
and  azimuth.  Humans  can  easily  distinguish  sounds  emanating  from  their  right  or  left 
because  the  signal  is  mainly  shifted  in  time  and  the  brain  uses  the  frequency  shift  from  right 
to  left  ear  to  determine  location.  To  determine  elevation  information  from  an  audio  signal, 
the  brain  uses  the  difference  in  frequency  attenuation  between  the  two  ears,  caused  by  the 
pinna  (i.e.,  the  shape  of  the  ear).  The  current  method  for  producing  virtual  localized  audio 
cues  involves  the  use  of  an  anechoic  chamber  with  speakers  located  at  various  elevations  and 
azimuths  (often  in  a  sphere  around  the  subject)  and  a  microphone  located  in  the  subject’s 
inner  ear.  The  microphone  is  used  to  record  the  frequency  attenuation  produced  by  the 
subject’s  pinna  for  each  speaker  location.  These  recordings  are  used  to  develop  what  is 
known  as  Head  Related  Transfer  Functions  (HRTFs)  which  are  later  used  to  produce  virtual 
localized  audio  signals  through  headphones.  When  using  this  method  the  HRTFs  developed 
are  specific  to  the  subject.  Generic  HRTFs  can  be  obtained  by  using  a  mannequin  in  lieu  of 
a  human  subject. 

Past  research  into  headphone-delivered  three-dimensional  audio  suggests  that  optimal 
localization  performance  results  from  the  combination  of  the  following  three  factors:  1) 
head-tracked  virtual  stimuli,  2)  synthesis  of  a  virtual  room,  and  3)  use  of  individualized  as 
opposed  to  generic  HRTFs.7  The  purpose  of  this  study  was  to  directly  compare  all  these 
factors  —  something  that  had  not  yet  been  done  in  a  single  study.  The  experiment  evaluated 
auditory  localization,  externalization  of  sound  images,  and  perceived  realism.  A  speech 
stimulus  was  given  with  three  levels  of  reverberation:  anechoic,  early  reflections,  and  full 
reverberation.  Both  individualized  and  non-individualized  HRTFs  were  studied  along  with  a 
continuously  updating  head  tracker  whose  information  was  used  to  update  the  position  of 
the  stimuli  or  was  disregarded.  Reverberation  was  found  to  have  a  significant  effect  on 
azimuth  and  elevation  errors.  Stimuli  which  include  reverberation  will  yield  lower  azimuth 
errors,  but  at  the  sacrifice  of  elevation  accuracy.  Head  tracking  and  individualized  HRTFs 
significandy  reduced  the  occurrence  of  reversals;  however,  there  was  no  other  clear 
advantage  in  using  individualized  HRTFs. 

In  1999  Bolia  et  al..8  conducted  an  experiment  to  evaluate  the  effectiveness  of  spatial  audio 
displays  on  target  acquisition  performance  under  three  scenarios:  no  spatial  audio,  free-field 
spatial  audio,  and  virtual  spatial  audio.  Subjects  performed  a  visual  search  task  with  and 
without  the  aid  of  spatial  audio.  Results  indicated  that  visual  search  times  decreased 
significantly  with  the  aid  of  free-field  or  virtual  spatial  audio.  The  subjects  sat  in  a  sphere 
where  at  the  apex  of  every  strut  there  existed  a  speaker  and  four  LEDs.  The  number  of 
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LEDs  lit  determined  whether  a  target  or  a  distracter  was  present.  Two  or  four  LEDs  lit 
meant  that  a  target  was  present,  while  one  and  three  LEDs  lit  meant  that  a  distracter  was 
present.  The  subject  responded  when  a  target  was  present  and  the  number  of  LEDs  lit  was 
two  or  four.  The  free-held  audio  performed  better  than  virtual  audio,  which  can  be 
attributed  to  defects  inherent  in  the  HRTF  collection  technique,  the  use  of  non- 
individualized  HRTFs,  and  the  spatial  resolution  of  the  HRTFs.  People  localize  well  in 
azimuth  with  non-individualized  transfer  functions,  but  they  make  fewer  front-back 
reflections  and  are  more  accurate  in  elevation  with  their  own  FIRTFs.  The  study 
demonstrated  a  reduction  in  search  time  by  a  factor  of  six  or  more  for  high-complexity 
searches  with  the  use  of  virtual  spatial  audio  cueing  without  a  corresponding  reduction  in  the 
accuracy  of  target  identification. 

Flanagan  et  al.. 9  combined  localized  audio  with  an  HMD  to  investigate  search  time  for  a 
visual  search  task  with  targets  outside  the  field  of  view  (FOV).  The  auditory  signal  was 
either  presented  at  the  beginning  or  continually  updated.  Three  scenarios  were  tested:  visual 
cue,  transient  audio  cue,  and  updating  audio  cue.  The  updating  audio  cue  was  more  effective 
than  the  transient  audio  cue  and  was  as  effective  as  the  visual  cue  in  reducing  search  time. 
Subjects  with  no  previous  HMD  experience  were  used.  The  transient  audio  was  three  short 
bursts,  while  the  updating  audio  was  a  single  noise  burst  every  eight  seconds.  The  visual 
display  was  a  sighting  circle  with  an  arrow  pointing  to  the  target.  The  participant  placed  a 
red  sighting  circle  over  the  white  target  for  0.5s.  The  ability  to  localize  elevation  is  poorer 
and  the  incidence  of  front-back  reflections  higher  when  listening  to  virtual  audio  than  under 
free-field  listening  conditions. 

Further  research  into  the  use  of  localized  audio  and  HMDs  for  visual  search  tasks  was 
conducted  by  Nelson  et  al..10  In  this  investigation  the  visual  displays  were  either  a  wide  FOV 
dome  or  an  HMD  and  were  accompanied  by  localized,  non-localized,  or  no  auditory 
information.  Localized  auditory  information  provided  significant  increases  in  target 
detection  performance  and  significant  reductions  in  workload  ratings  as  compared  to  no 
auditory  or  non-localized  audio.  Localized  audio  resulted  in  more  efficient  search  strategies. 
Targets  approached  from  outside  the  field  of  view  from  a  random  direction.  The 
participants  pressed  a  mouse  button  when  they  detected  a  target.  The  participant  also 
marked  the  location  of  the  target  with  a  head-slaved  cursor.  Subjects  completed  the  NASA 
Task  Load  Index  for  workload.  All  metrics  of  performance  —  efficiency;  workload,  and  head 
motion  -  revealed  a  significant  advantage  for  conditions  in  which  localized  auditory  cues 
were  provided.  The  reduction  in  head  motion  and  velocity  may  improve  pilot  performance 
in  high-G  environments. 

Visual  search  tasks  were  not  the  first  use  for  localized  audio;  Begault  and  Pittman11  used 
localized  audio  to  improve  upon  the  current  Terrain  Collision  Avoidance  System  (TCAS).  In 
their  study  they  compared  the  current  head-down  TCAS  system  to  a  head-up  audio  TCAS 
system  by  measuring  the  time  needed  to  capture  visual  targets.  Ten  commercial  airline  crews 
were  tested  under  full-mission  simulation  conditions  and  were  given  either  the  standard 
visual-audio  TCAS  advisory  or  a  3-D  aural  advisory  (the  3-D  aural  advisory  did  not  include  a 
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map  display).  The  target  remained  at  a  fixed  distance  and  speed  relative  to  the  subjects  to 
eliminate  differences  between  crews  as  a  function  of  aircraft  movement.  The  aural  alert 
consisted  of  a  non-speech  pre-advisory  and  a  voiced  “TRAFFIC-TRAFFIC”.  The  elevation 
cues  were  exaggerated  because  elevation  judgments  are  often  compressed  relative  to  the 
actual  target  positions  when  listening  through  non-individualized  FIRTFs.  Flowever  azimuth 
was  not  exaggerated.  The  results  show  that  the  use  of  3-D  audio  significantly  reduced  the 
acquisition  time  by  500  ms.  The  results  were  comparable  to  past  research  done  by  Begault, 
and  there  appear  to  be  no  significant  benefits  of  exaggerating  the  stimuli  in  azimuth. 

Begault  followed  up  on  the  previous  experiment  by  adding  3-D  audio  to  the  visual  display 
instead  of  replacing  it.12  Each  crew  flew  the  route  twice,  once  with  the  addition  of  3-D 
audio  and  once  without.  The  results  of  this  experiment  agree  with  the  past  experiments  in 
that  the  addition  of  3-D  audio  helped  reduce  target  acquisition  time  (225-749  msec).  The 
experiment  was  done  under  the  same  condition  as  the  previous  experiment.  Though  the 
improvement  in  time  is  not  great,  the  workload  was  relatively  light,  and  it  could  be  that  in  a 
high-workload  situation  one  would  see  greater  benefits  of  using  3-D  audio  cues.  One 
interesting  thing  that  was  found  in  the  experiment  is  that  pilots  did  not  utilize  the  elevation 
cue  effectively,  presumably  because  the  successful  perception  of  elevation  cues  from  FIRTFs 
is  more  difficult  than  azimuth.  The  suggestion  made  is  that  the  aural  stimuli  should  include 
location,  for  example  “TRAFFIC,  HIGH.” 

Combining  localized  audio  and  visual  displays  have  proven  to  be  effective  in  improving 
performance  for  the  TCAS.  Simpson  et  al.13  looked  at  the  performance  benefits  of  a 
combined  visual  and  audio  display  in  an  experiment  where  the  subjects  were  required  to 
acquire  and  identify  a  target  under  four  conditions:  1)  no  display;  2)  a  visual  display 
combined  with  a  non-spatialized  warning  sound;  3)  a  visual  display  combined  with  a  clock- 
coordinate  speech  signal;  and  4)  a  visual  display  combined  with  a  spatialized  auditory 
warning  sound.  The  subjects  participated  in  a  fully-immersive  simulated  flight  task.  For  the 
no-display  condition,  subjects  had  to  search  without  any  aid,  while  in  the  other  three 
conditions  a  Traffic  Advisory  System  (TAS)  provided  the  subject  with  information  about  the 
presence,  relative  altitude,  and  direction  of  the  target.  For  the  visual  with  clock- audio 
condition,  the  TAS  system  was  augmented  by  a  non-spatialized  audio  cue  indicating  clock 
direction,  elevation,  and  distance  of  the  target,  for  example  “traffic,  9  o’clock  high,  1  mile.” 
In  the  TAS  with  3D  audio  condition,  the  visual  TAS  display  was  supplemented  with  a 
spatialized  audio  cue  that  consisted  of  a  chirp  signal  and  a  verbal  cue  indicating  the  relative 
elevation  of  the  target  (e.g.,  “traffic,  low”).  The  distance  corresponded  to  the  vocal  effort  of 
the  stimuli,  with  shouted  speech  indicating  distant  targets  and  conversational  speech 
indicating  close  targets.  The  target  aircraft  was  on  a  head-on  collision  path  with  the  subject. 
No  collisions  occurred  in  the  TAS-clock  and  TAS-3D  audio  conditions.  The  data  show  that 
the  TAS-3D  audio  always  led  to  the  fastest  response  times  and  that  the  advantage  of  TAS- 
3D  audio  was  greater  when  the  targets  were  located  in  the  rear  hemisphere. 

This  past  research  has  demonstrated  the  utility  of  virtual  3-D  auditory  displays  for  tasks  that 
are  relevant  to  air  combat:  1)  enhancing  visual  target  detection  and  identification;  2) 
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increasing  the  effectiveness  of  collision  avoidance  displays;  3)  improving  the  monitoring  of 
simultaneous  communication  signals;  and  4)  alleviating  problems  associated  with  visual 
detection  tasks  in  helmet-mounted  displays.  However  no  study  has  yet  to  address  the 
effectiveness  of  3D  auditory  displays  under  high  levels  of  sustained  acceleration.  Nelson  et 
al.14  evaluated  the  ability  to  localize  a  virtual  audio  source  during  exposure  to  1.0,  1.6,  2.5, 

4.0,  5.6,  and  7.0  +GZ  for  auditory  cues  along  the  horizontal  plane  (elevation  of  0°).  The 
Dynamic  Environment  Simulator  was  used  to  simulate  the  acceleration  stresses  often 
encountered  by  pilots.  The  subjects  used  a  right-hand  knob  to  rotate  a  radius  vector  to  the 
perceived  azimuth.  There  were  an  equal  amount  of  cues  for  each  subject  in  each  spatial 
quadrant  of  the  horizontal  plane.  Both  the  average  localization  error  and  the  percentage  of 
reversal  confusions  indicated  that  increase  +GZ  level  did  not  correlate  to  reduction  in  ability 
to  localize  in  the  horizontal  plane.  It  wasn’t  until  subjects  experienced  7.0  +GZ  that  any 
significant  increases  in  average  localization  error  were  found.  Thus  it  appears  that  the  use  of 
3-D  audio  displays  will  not  be  compromised  by  conditions  of  high  acceleration,  which  are 
often  experienced  during  tactical  air  combat  environments. 

The  research  conclusions  from  the  localized  audio  literature  are:  1)  Three-dimensional 
auditory  cues  can  reduce  the  time  it  takes  to  acquire  targets  in  space  with  no  apparent 
reduction  in  the  accuracy  of  target  acquisition.  2)  Humans  can  localize  better  in  azimuth 
than  in  elevation  and  degradation  in  elevation  increases  with  the  use  of  non-individualized 
HRTFs.  3)  Greater  reductions  in  time  were  found  when  the  auditory  cue  was  continuously 
updated  as  opposed  to  transiently;  however,  no  comparisons  were  made  between 
continuously  updating  and  human  speech.  4)  Localized  audio  resulted  in  more  efficient 
search  strategies,  meaning  less  head  movement.  5)  Non-individualized  HRTFs  led  to  an 
increase  in  front-back  reversals  compared  to  individualized  HRTFs.  6)  A  subject’s  ability  to 
localize  sound  is  not  degraded  under  high  accelerations. 
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METHODS 


It  is  the  objective  of  this  study  to  investigate  the  combined  benefits  of  the  NDFR  symbology 
and  three-dimensional  audio  cues  for  target  acquisition  tasks  in  a  stationary  simulator  using  a 
helmet-mounted  display.  Past  research  has  shown  that  use  of  HMDs  leads  to  more  time 
spent  looking  off-boresight  and  less  time  referencing  cockpit  displays  without  any  change  in 
performance.  Three-dimensional  audio  research  has  demonstrated  that  the  use  of  a 
combined  visual/ audio  system  can  decrease  subject  reaction  time  and  increase  performance. 
Visual-audio  systems  performed  better  than  visual  or  audio  systems  alone.  In  this  study 
subjects  flew  in  a  simulator  while  looking  for  targets  using  visual,  audio,  or  multimodal 
visual-audio  cues.  The  effectiveness  of  the  NDFR  in  providing  attitude  information  quickly 
and  succinctly  will  be  compared  to  the  combined  target  cueing  systems  ability  to  reduce 
visual  search  time. 

Participants 

Twenty  volunteers  consisting  of  male  and  female  MIT  affiliates  participated  in  this 
experiment.  The  volunteers  had  adequate  eyesight  and  no  significant  hearing  impairments. 
The  subjects  ranged  from  experienced  pilots  to  people  with  no  prior  flying  experience. 

Experiment 

The  subjects  flew  an  aircraft  in  a  desktop  simulator  while  acquiring  targets  in  as  short  a  time 
as  possible.  Two  independent  variables  were  manipulated.  The  first  was  the  aircraft  attitude 
symbology  set.  The  two  symbologies  utilized  were  Military  Standard  (Mil-Std)  symbology 
and  the  Non-Distributed  Flight  Reference  (NDFR)  symbology.  The  second  independent 
variable  was  the  target  location  cueing  type.  The  three  levels  of  cueing  conditions  were 
audio  only,  visual  only,  and  combined  audio/ visual.  The  six  resulting  conditions  are  shown 
in  Table  1. 


Attitude  Displays 

Target  Cues 

Military  Standard 

Visual  Only 

Auditory  Only 

Combined  Visual/ Audio 

Non-Distributed  Flight  Reference 

Visual  Only 

Auditory  Only 

Combined  Visual/ Audio 

Table  1:  Experiment  Conditions 


The  dependent  variables  were  search  time  and  flight  performance.  Search  time  is  defined  as 
the  time  it  takes  the  subject  to  bring  the  target  within  the  helmet-mounted  aim-sight  reticle 
from  trial  initiation.  Flight  performance  was  determined  by  the  subject’s  deviation  from  a 
commanded  heading  and  altitude. 
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The  subjects  were  presented  with  targets  from  15  locations.  Half  of  the  subjects  began  the 
trials  with  the  Mil-Std  attitude  display  and  the  other  half  with  the  NDFR  attitude  display. 
Each  subject  saw  every  experimental  condition.  The  cueing  type  was  pseudo-randomized 
across  attitude  display. 

The  targets  originated  from  azimuths  of  0°,  ±30°,  and  ±60°;  with  elevations  of  0°,  ±45°. 
These  target  locations  were  repeated  for  each  trial.  The  location  of  the  target  was  fixed  to  the 
x-axis  (nose-to-tail)  of  the  subject’s  aircraft  and  did  not  change  when  the  subjects  moved 
their  heads.  There  were  15  target  locations  present  for  each  targeting  cue  and  the  targets 
were  repeated.  Each  cue  was  given  for  all  15  target  locations.  Thus  for  a  given  attitude 
display  the  subject  saw  90  targets,  30  for  each  of  the  three  target  cueing  types  {15  (target 
locations)  x  2  (for  repetition)  x  3  (for  cueing  type) } .  A  Latin  square  was  used  to  determine 
the  order  of  the  cueing  type.  The  target  locations  were  randomized  and  the  repetition  is  just 
the  reverse  order  of  the  original  combinations  of  target  location  and  cueing  type.  Between 
each  real  target,  a  dummy  target  appeared  directly  in  front  of  the  simulated  aircraft.  This 
forced  the  subjects  to  return  their  head  to  boresight  between  each  target,  ensuring  that  every 
target  search  began  with  the  subject  looking  straight  ahead.  The  total  number  of  targets, 
including  dummy  targets,  seen  by  a  subject  throughout  the  experiment  was  360. 

Apparatus 

The  study  took  place  at  the  Man  Vehicle  Laboratory,  MIT,  in  Cambridge,  MA.  The 
simulation  was  displayed  within  a  CyVisor  HMD,  which  allows  for  a  19°  by  24°  field  of  view. 
The  subjects  were  seated  in  a  room  with  the  examiner  and  had  a  joystick  to  fly  the  simulator. 
Outside  noise  was  minimized  to  limit  any  confusion  with  the  auditory  signals.  A  software 
package  called  SLAB  provided  the  auditory  cues.  Non-individualized  HRTFs  were  used.  A 
HMD  with  a  Flock  of  Birds  magnetic  head  tracker  provided  both  the  attitude  displays  and 
the  visual  cues. 

Symbology  Sets 

Military  Standard  Head-Up  Display  04  LIT 

The  Military  Standard  HUD,  depicted  in  Figure  5,  was  used  as  a  baseline  comparison  with 
the  NDFR  display.  It  includes  a  climb/ dive  ladder  with  flight  path  marker  for  attitude 
reference,  an  airspeed  indicator  (left,  digital  readout  with  dial  and  counterpointer),  an 
altimeter  (right,  digital  readout  with  dial  and  counterpointer),  and  a  heading  tape  centered  at 
the  top  of  the  screen. 


12 


26  27  28  29 

i  .  I  i  i  .  I 

1 

/ 

1 1  if 

^  10  049 

G  1121 

M  1.72 
-  £3t 7 
«  0.0 

T 

VV  5 

1  500  LAW 

- 

^tT+220 

A  9500 

Figure  5:  Military  Standard  Display.  In  this  illustration  the  simulated  aircraft  is  rolled 
right  less  than  five  degrees,  pitched  up  two  degrees,  with  the  airspeed  at  1,119  knots,  an 
altitude  of  10,049  ft,  and  a  heading  of  276  degrees. 


Non-Distributed  Flight  Reference 


The  NDFR  was  developed  by  the  Air  Force 
Research  Laboratory  as  a  flight  reference  that  has  less 
visual  occlusion  than  other  modem  symbol  sets.  The 
NDFR  makes  use  of  the  “orange  peel”  arc  for 
attitude  information.  Airspeed,  heading,  and  altitude 
are  displayed  around  a  flight  path  marker  to  give  the 
pilot  all  of  the  ownship  status  information  in  a  small, 
well-defined  area.  The  NDFR  was  flight  tested  in 
2001  aboard  a  VISTA  F-16  equipped  with  the  Viper 
Mark-IV  40-degree  monocular  FOV  HMD.6  The 
recommended  changes  resulted  in  the  development 
of  the  advanced  display,  a  version  of  which  was  used 
in  this  study.  The  NDFR  display  used  in  the 
experiment  is  shown  in  Figure  6. 


Figure  6:  NDFR  Display.  In 

this  figure,  the  simulated  aircraft  is 
rolled  left  30°,  pitched  up  2°,  with 
an  airspeed  of  634  knots,  an 
altitude  of  6,370  ft,  and  heading 
north  at  3°. 
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Visual  Cue 


The  visual  cue  consists  of  a  single  circle  with  a  pointer  centered  on  the  aim-sight  reticle 
which  is  located  in  the  center  of  the  field-of-view.  The  pointer  shows  where  the  subjects 
should  move  their  head  to  obtain  visual  contact  with  the  target  aircraft.  The  visual  cue  within 
the  aim-sight  reticle  is  shown  in  Figure  7,  in  which  the  target  is  down  and  to  the  right  of  the 
subject’s  current  helmet  boresight. 


Figure  7:  Visual  Cue 


Auditory  Cue 

The  auditory  cue  was  a  female  voiced  message,  “Target-Target,”  provided  through 
headphones  using  the  SLAB  software  developed  by  NASA,  which  is  available  to  the  public. 
The  cue  emanated  from  the  same  azimuth  and  elevation  as  the  visual  target.  The  voiced  cue 
had  poor  resolution  in  elevation  since  non-individualized  HRTFs  were  used.  The  cue  ran 
continuously  at  a  constant  frequency,  pitch,  and  volume,  and  would  stop  only  after  the 
subject  visually  acquired  the  target. 
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Procedure 


The  subjects  were  given  a  set  of  instructions  including  a  description  of  the  experiment  and  a 
set  of  instructions  as  to  how  to  interpret  the  symbology  sets.  They  were  told  that  their 
objective  was  to  visually  acquire  a  set  of  targets  while  maintaining  a  desired  heading  and 
altitude.  The  flying  task  was  simplified  by  minimizing  the  number  of  controls  needed  to 
maintain  heading  and  altitude.  The  subjects  had  no  rudder  pedals  and  thrust  was  held 
constant  throughout  the  entire  experiment.  The  subjects  were  seated  and  fitted  with  the  3-D 
auditory  headphones,  the  HMD,  and  the  head  tracker.  Before  the  recorded  trials  began  they 
were  allowed  to  fly  the  simulator  with  the  different  attitude  displays. 

The  experiment  began  after  the  subject  felt  comfortable  with  both  attitude  displays.  The 
subjects  did  not  have  any  practice  with  the  target  cues.  They  began  the  trial  at  the  desired 
altitude  and  heading,  and  were  asked  to  maintain  500  ft  for  altitude  and  20  for  heading  using 
joystick  inputs.  The  experiment  included  simulated  turbulence.  While  maintaining  heading 
and  altitude,  the  subject  was  required  to  move  his/her  head  in  order  to  place  the  target 
within  the  aim-sight  reticle,  i.e.,  keep  the  target  within  five  degrees  of  the  center  of  their 
field-of-view.  Once  the  subject  acquired  the  target  i.e.,  maintained  visual  contact  with  it  for 
three  seconds,  the  target  disappeared  and  a  new  target  appeared.  Throughout  the  trials  the 
subject  saw  a  simulated  world  that  included  mountains,  fields,  and  water.  The  subjects  all 
flew  the  same  flight  profile  under  the  same  weather  and  turbulence  conditions.  After  the 
subjects  completed  the  trial  they  were  asked  to  fill  out  a  free  response  subjective 
questionnaire.  The  entire  experiment  lasted  approximately  1.5  hours  with  one  hour  of  actual 
simulated  flying.  An  explanation  of  the  data  analysis  done  is  included  in  the  next  section. 
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RESULTS 


The  repeated-measures  General  Linear  Model  (GLM)  Analysis  of  Variance  (ANOVA)  was 
used  to  determine  significant  main  and  cross  effects  at  the  p  <  0.05  level;  with  a  Huynh-Feldt 
correction  applied  to  the  p-values.  Statistical  analysis  was  performed  using  SYSTAT-11  and 
StatXact-5  Cytel  software.  The  GLM  was  used  to  determine  the  effects  of  attitude  display, 
target  cue,  target  location,  repetition,  attitude  display  order,  and  flying  experience  on  both 
flight  performance  and  search  time.  Two  metrics  were  used  to  determine  flight 
performance:  heading  (degrees)  and  altitude  (feet)  as  deviations  from  the  directed  heading 
and  altitude.  Search  time  was  determined  as  the  amount  of  time  the  subjects  took  to  visually 
acquire  each  target  by  holding  it  within  the  aim-sight  reticle  for  three  seconds. 

Twenty  subjects  participated  in  the  experiment.  Three  were  removed  because  they  did  not 
finish  the  experiment,  and  two  others  were  removed  later  from  the  analysis  because  their 
data  were  drastically  unlike  the  other  15  subjects. 

Flight  Performance  Results 

The  flight  performance  metrics  are  the  root  mean  square  errors  (RMSE)  from  the 
commanded  heading  and  altitude,  as  shown  in  Equation  1,  where  the  flight  metric,  Fm,  is 
subtracted  from  the  commanded  value. 


RMSE  = 


Eq.  1 


For  this  experiment  the  commanded  heading  was  220°  and  the  altitude  was  1 1,500  ft.  For 
both  heading  and  altitude  RMSE  there  was  great  variation  among  subjects.  No  significant  (p 
<  0.05)  main  effects  were  found  by  SYSTAT  GLM  ANOVA.  As  expected,  since  the  display 
was  helmet-mounted,  target  cue,  target  location,  and  repetition  did  not  have  any  significant 
main  or  cross  effects  with  heading  and  altitude  deviation.  Surprisingly,  subject  flight 
performance  did  not  change  significantly  between  the  Mil-Std  and  the  NDFR  displays. 
Figures  8  and  9  show  the  average  heading  and  altitude  RMSE  for  all  subjects,  where  attitude 
display  (1)  represents  Mil-Std  symbology  data,  and  (2)  represents  NDRF  results. 
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Figure  8:  Effect  of  Attitude  Display 
Altitude  RMSE 


Figure  9:  Effect  of  Attitude  Display  on 
on  Heading  RMSE. 


The  overall  mean  for  heading  RMSE  was  10.502  ±0.356°  for  the  Mil-Std  display  and  15.675 
±0.531°  for  the  NDFR  display.  The  overall  mean  for  altitude  RMSE  was  812.934  ±19.846 
(ft)  for  the  Mil-Std  display  and  1214.156  ±34.932  (ft)  for  the  NDFR  display.  Though  these 
averages  seem  significantly  different  they  include  all  other  effects  and  are  vulnerable  to 
outliers  in  the  data. 


Figures  10  and  1 1  show  the  heading  and  altitude  RMSE  for  a  single  subject.  Throughout  the 
experiment  the  subjects  saw  a  total  of  180  targets,  90  for  each  attitude  display.  This  target 
number  (1-180)  is  displayed  on  the  x-axis.  Figures  10  and  11  show  the  data  for  a  subject 
using  Order  1:  Mil-Std  then  NDFR  display.  For  this  subject  attitude  display  had  no  effect  on 
flight  performance.  The  variation  for  the  first  90  targets  (Mil-Std)  does  not  differ  from  the 
second  90  targets  (NDFR).  Short  of  a  few  outlying  points,  this  trend  is  consistent  across  all 
subjects.  There  existed  a  significant  cross  effect  between  attitude  display,  target  location, 
target  cue,  and  order  (p  —  0.026).  When  looking  closely  at  the  data,  this  effect  is  the  result  of 
random  outliers  in  the  data. 
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There  were  no  significant  main  or  cross  effects  of  flying  experience  on  either  heading  or 
altitude  deviation.  Both  are  plotted  below  in  Figure  12.  Six  subjects  had  flying  experience, 
four  had  less  than  5  hours  and  the  other  two  had  40  and  170  hours.  The  single  subject  with 
170  hours  was  the  only  licensed  pilot.  There  was  no  difference  in  flight  performance  among 
the  subjects  with  5  hours  or  less  of  experience.  The  subject  with  170  hours  consistently  had 
one  of  the  lowest  heading  and  altitude  RMSE  scores  and  one  of  the  smallest  differences 
between  attitude  displays.  There  also  existed  no  difference  in  performance  or  preference 
between  the  two  attitude  displays  as  a  function  of  flying  experience.  Subject  attitude  display 
preference  is  discussed  in  further  detail  in  Chapter  5. 


Figure  12:  Effect  of  Flying  Experience  on  Heading  and  Altitude  RMSE 
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Search  Time  Results 


The  significant  main  and  cross  effects  found  for  time-to-target,  using  the  repeated  measures 
GLM  ANOVA,  are  listed  in  Table  2.  The  first  column  in  the  table  lists  the  source  of  the 
significant  effect,  and  the  second  column  lists  the  F-value  (F),  degrees  of  freedom  (df),  and 
p-value. 


Source 

(F,  df,  p-value) 

Attitude  Display 

(21.116, 1,0.001*) 

Attitude  Display*Order 

(21.170,  1,  0.00F) 

Attitude  Display* Azimuth 

(4.578,  4,  0.003) 

Attitude  Display*  Azimuth*Order 

(4.551,4,  0.003) 

Elevation*  Azimuth 

(3.178,  8,  0.004) 

Elevation*  Azimuth*Order 

(2.886,  8,  0.008) 

Elevation*Rep*Order 

(5.248,  2,  0.013) 

Attitude  Display*Elevation*Rep 

(3.529,  2,  0.045) 

Attitude  Display* Azimuth*Repetition* Order 

(2.864,  4,  0.033) 

Attitude  Display*Target  Cue*Repetition 

(7.256,  2,  0.003) 

Attitude  Display*Target  Cue*Repetition*Order 

(9.381,2,  0.001) 

Attitude  Display*Elevation* Azimuth* 
Target  Cue*Repetition*Order 

(2.270,  16,  0.016) 

Table  2:  GLM  Search  Time  p  <  0.05  Results 


The  only  significant  main  effect  found  was  that  of  Attitude  Display,  (p  —  0.001).  This  main 
effect  is  plotted  in  Figure  13  for  both  between-  and  within-  subjects.  Attitude  Display  (1)  is 
the  Mil-Std  display  while  (2)  is  the  NDFR  display.  The  difference  in  search  time  varied  from 
subject  to  subject  (right-hand  plot  below  and  Table  3).  Not  all  subjects  performed  better 
with  the  NDFR  display. 


i.  Not  Huynh-Feldt  corrected,  df  =  1 
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ATT  DISPLAY  subject 

Figure  13:  Effect  of  Attitude  Display  on  Search  Time  (between,  within  subjects).  The 

y-axis  consists  of  the  time-to-target  in  seconds.  Attitude  Display  (1)  =  Mil-Std;  (2)  =  NDFR. 

Table  3  shows  the  averages  and  differences  between  attitude  displays  on  time-to-target  in 
seconds  for  every  subject.  The  difference  noted  is  Military  Standard  minus  NDFR;  thus, 
positive  values  indicate  a  reduction  in  time  when  using  the  NDFR  display.  The  significant 
effect  of  attitude  display  on  search  time  is  attributable  to  a  few  subjects  and  other  effects, 
e.g.,  Subject  1 1  and  Order. 
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(sec) 


EBHEI 

Difference 

0 

5.83 

5.52 

0.31 

i 

7.18 

7.60 

-0.41 

5 

8.91 

7.88 

1.03 

7 

10.51 

9.59 

0.91 

8 

7.71 

8.60 

-0.89 

9 

8.30 

7.02 

1.27 

10 

9.98 

11.51 

-1.52 

11 

12.70 

8.18 

4.52 

12 

9.48 

12.32 

-2.84 

13 

7.99 

6.94 

1.06 

16 

8.07 

10.05 

-1.98 

17 

8.67 

6.46 

2.22 

18 

6.95 

7.36 

-0.41 

19 

12.41 

10.71 

1.70 

20 

9.00 

11.86 

-2.86 

Overall 

8.91 

8.77 

0.14 

Table  3:  Time-to-Target  Averages  between 
Attitude  Displays  (sec)  SE  =  ±0.15 

The  cross  effect,  Attitude  Display  and  Order,  was  also  found  to  be  significant.  Figure  10 
shows  the  overall  effect  of  Attitude  Display  and  Order,  while  Figure  1 1  shows  the  difference 
in  search  time  between  the  Mil-Std  and  NDFR  displays  for  each  subject.  Order  (1)  refers  to 
sequence  Mil-Std:NDFR,  while  (2)  refers  to  NDFR:Mil-Std. 
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Figure  14:  Cross  Effect  of  Attitude  Display  and  Order  (1,2)  =  ({Mil- 
Std:NDFR},{NDFR:Mil-Std}).  The  solid  bar  =  Mil-Std,  hash  bar  =  NDFR. 
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SUBJECT 

Figure  15:  Cross  Effect  of  Attitude  Display  and  Order.  Y-axis  is  the  difference  in 
search  time  between  Mil-Std  and  NDFR  (i.e.,  Mil-Std  -  NDFR).  Shape  depicts 
Order  (1,2)  =  (O,  X). 

The  significant  cross  effect  of  Attitude  Display  *  Order  (p  —  0.001)  is  consistent  with  a 
learning  effect.  The  subjects  always  performed  better  with  the  second  display  presented. 
Figure  1 1  shows  that  the  difference  between  displays  is  smaller  for  subjects  who  started  the 
simulation  with  the  Mil-Std  display  and  ended  with  the  NDFR.  The  average  difference 
between  the  first  display  and  the  second  for  Order  (1)  is  1.401  ±0.2  s,  while  the  average  for 
Order  (2)  is  1.745  ±0.35  s.  From  this  point  the  cross  effect  of  Attitude  Display  *  Order  (as  it 
is  called  in  the  analysis)  will  be  referred  to  as  the  First  Display  effect. 

Two  cross  effects,  (1)  Attitude  Display  *  Azimuth,  and  (2)  First  Display  *  Azimuth,  were 
found  to  be  significant  at p  —  0.003.  The  plots  below  show  the  combined  effects  of  these 
factors  within  each  subject.  The  x-axis  is  the  target  azimuth  and  ranges  from  0-4  (-60°,  -30°, 
0°,  +30°,  +60°).  The  y-axis  shows  the  difference  between  the  NDFR  and  Mil-Std  displays. 
The  order  is  coded  into  the  subject  number  which  is  located  on  top  of  the  plot.  Subjects 
with  odd  numbers  flew  the  simulation  with  Order  1  (Mil-Std:NDFR),  while  those  with  even 
numbers  flew  with  Order  2  (NDFR:Mil-Std).  The  biggest  difference  between  the  NDFR 
and  the  Mil-Std  display  occurs  at  Azimuth  3  (+30°).  The  difference  is  negative  for  subjects 
in  Order  (1)  and  positive  for  Order  (2).  This  is  similar  to  the  effect  seen  above  in  that  the 
second  display  resulted  in  faster  search  times.  The  result  for  Azimuth  3  is  significantly 
different  (p  —  0.008)  from  the  average  of  the  other  four  Azimuths,  0-2,  and  4. 
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The  effect  of  target  location  (embracing  both  elevation  and  azimuth)  and  target  location  * 
order  were  found  to  be  significant  (p  —  0.004,  0.008).  Figure  15  shows  the  average  effect, 
across  all  subjects,  of  target  location  on  search  time.  The  target  locations  are  separated  into 
three  groups  of  five  representing  the  three  elevations  (+40°,  0°,  -40°)  and  five  azimuths  (-60°, 
-30°,  0°,  +30°,  +60°).  The  vertical  lines  in  the  plot  below  denote  the  three  elevation  angles 
+40°,  0°,  and  -40°.  Within  each  elevation  group  the  azimuths  range  from  -60°  to  +60°.  No 
significant  main  effects  were  found  on  time-to-target  for  azimuth  or  elevation.  There  is  a 
significant  difference  between  the  target  locations  at  0°  elevation  and  ±40°,  but  no  difference 
between  +40°  and  -40°  elevations  (p  —  0.0005).  Order  2  gave  slower  search  times  than 
Order  1  for  almost  all  target  locations. 

Elevation  had  a  significant  cross  effect  with  Repetition  and  Order  (p  —  0.013)  as  well  as 
Repetition  and  Attitude  Display  (p  —  0.045)  on  search  time.  The  difference  between  the  first 
and  second  repetitions  at  an  elevation  of  -40°  for  Order  (2)  was  greater  than  the  rest,  3.874 

±0.591  s.  The  difference  between 
repetitions  was  also  greater  at  the  same 
elevation  for  the  Mil-Std  display,  3.367 
±0.488  s.  In  both  cases  the  second 
repetition  had  a  faster  search  time. 
Figure  16  shows  the  cross  effect  of 
Elevation,  Repetition,  and  Order  on 
search  time.  The  x-axis  represents  the 
elevations  (+40°,  0°,  -40°)  and  the  y-axis 
shows  the  difference  between  the 
second  and  first  repetitions. 

0  4  8  1  2  1  6  r 

TGT_LOCATION 

Figure  17:  Effect  of  Target  Location  on  Search  Time 
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Figure  18:  Cross  Effect  of  Elevation,  Repetition,  and  Order  on  Search  Time 
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A  repeated-measures  GLM  AN OVA 
analysis  found  no  significant  main  effect 
target  cue  on  search  time.  Figure  17, 
however,  shows  the  effect  of  target  cue 
search  time  and  the  trend  is  significant  \p 
0.00005  by  the  non-parametric  Page  Test 
(StatXact-5,  Cytel),  from  slowest  to 
fastest  search  times:  Audio  Only  (10.260 
±0.221  s),  Visual  Only  (8.772  ±0.178s), 
Combined  Visual/ Audio  (7.497 
±0.128s)],  and  holds  for  all  but  two  of 
subjects,  0  and  8. 

The  last  four  seemingly  significant  cross 
effects,  or  (1)  Target  Cue  *  Repetition  * 

Attitude  Display  (p  —  0.003);  (2)  Target  Cue 
*  Repetition  *  First  Display  (p  —  0.001);  (3) 
First  Display  *  Azimuth  *  Repetition  (p  — 
0.033);  and  (4)  First  Display  *  Elevation  * 
Azimuth  *  Target  Cue  *  Repetition  (p  — 

0.01 6)  can  be  attributed  to  a  few  outliers  in 
the  data. 
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Figure  19:  Box  plots  depicting  effect  of  Target 
Cue  on  Search  Time.  The  box  plots  show  the 
median  (line  through  box),  25%  quartile  (bottom  of 
box),  and  75%  quartile  (top  of  box);  data  that  are 
more  than  three  standard  deviations  from  the 
average  are  shown  outlying  points  beyond  the  legs. 
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DISCUSSION 


The  results  of  this  experiment  showed  that  there  was  no  difference  in  flight  performance 
between  the  Military  Standard  and  Non-Distributed  Flight  Reference  displays.  There  existed 
a  strong  learning  effect  on  search  times  affected  by  the  order  in  which  the  attitude  displays 
were  presented.  It  was  easier  for  subjects  to  transition  from  the  Military  Standard  to  the 
NDFR  than  in  the  opposite  order.  The  greatest  difference  between  attitude  displays  on 
search  time  existed  at  an  azimuth  of  +30°,  while  the  fastest  search  times  occurred  at  target 
locations  within  the  0°  elevation  angle.  The  difference  between  repetitions  was  greatest  at  an 
elevation  of  0°,  with  the  second  order,  and  Mil-Std  display.  Target  cue  had  a  significant 
effect  on  search  time  with  the  audio  only  condition  the  slowest,  then  the  visual  only 
condition,  and  the  combined  condition  the  fastest. 

This  experiment  finds  no  effect  of  display  (Military  Standard  vs.  Non-Distributed  Flight 
Reference)  on  heading  and  altitude  RMSE.  This  result  is  consistent  with  the  experiments  of 
Geiselmen  et  al.2  and  Jenkins  et  al.4  where  no  difference  in  flight  performance  was  found 
when  using  the  NDFR  display.  Between  this  experiment  and  past  research  the  NDFR  is 
proving  to  be  an  effective  attitude  display  for  visual  search  tasks.  It  was  viewed  by  the 
subjects  to  be  a  “cleaner  and  more  integrated  [display,  but]  more  challenging  to  use.”  It 
provided  attitude  information  quickly,  but  took  the  subjects  longer  to  become  accustomed 
to  using  it,  translating  the  arc  into  pitch  and  bank  information.  Some  subjects  complained 
that  the  Military  Standard  display  contained  too  many  moving  segments  that  took  up  much 
of  the  visual  space,  distracting  them  from  finding  the  targets.  In  the  end  the  NDFR  display 
takes  more  time  to  become  accustomed  to  but  provides  adequate  information  in  a  clean  and 
integrated  display,  while  the  Mil-Std  is  more  intuitive  when  controlling  pitch  and  bank  but 
requires  a  longer  visual  scan  to  obtain  necessary  attitude  information.  Future  work  should 
focus  on  improving  the  NDFR’s  ability  to  display  bank  and  pitch  information.  Two  possible 
improvements  include  broadening  and/ or  lengthening  the  0°  pitch  lines  and  including  a 
horizon  line  at  0°  pitch. 

The  time-to-target  was  affected  by  both  the  attitude  display  used  and  the  order  in  which  it 
was  presented.  The  longer  visual  scan  required  by  the  Mil-Std  display  may  explain  why  use 
of  the  NDFR  display  resulted  in  faster  search  times.  Subjects  who  were  presented  with  the 
attitude  displays  in  the  second  order  (NDFR:Mil-Std)  had  a  greater  reduction  in  search  time. 
The  simulated  aircraft  was  easier  to  control  with  the  Mil-Std  display.  Thus,  the  subjects 
seeing  that  display  second  had  an  easier  display  to  work  with  when  they  were  more  familiar 
with  the  experiment. 

The  difference  between  the  second  and  first  attitude  displays  was  found  to  be  most  sensitive 
at  the  +30°  azimuth.  The  subjects  had  the  greatest  reduction  in  search  time  for  the  faster 
display  at  this  azimuth.  Although  it  is  possible  that  subjects  began  their  visual  searches  to 
the  right,  more  analysis  is  needed  to  resolve  this  apparent  preference  in  azimuth. 
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It  comes  as  no  surprise  that  the  fastest  search  times  occurred  at  0°  elevation  (p  <  0.0005), 
closest  to  the  center  of  the  subject’s  FOV.  Another  possible  reason  for  this  finding  is  that 
the  audio  cue  for  these  targets  was  not  required  to  provide  any  up-down  information.  This 
poor  resolution  in  elevation  for  the  audio  cue  will  be  discussed  later.  The  first  repetitions  for 
targets  located  at  -40°  elevation  were  found  to  have  the  greatest  search  times  (p  <  0.05).  At 
the  start  of  the  experiment,  subjects  did  not  tilt  their  head  down  enough  to  acquire  the 
lowest  targets  efficiently  during  their  visual  search.  Once  the  subjects  became  accustomed  to 
the  target  location  extremes  the  search  time  for  targets  located  below  their  horizon 
decreased  significantly. 

This  study  found  a  significant  effect  of  target  cue  on  search  times.  The  three  target  cues 
studied  were:  (1)  3-D  audio  only,  a  voiced  signal  =  “Target-Target”;  (2)  visual  only,  a  line 
emanating  from  aim-sight  reticle  pointing  towards  target  aircraft;  and  (3)  a  combined 
audio/visual  cue,  providing  both  cues.  Use  of  the  3-D  audio  only  cue  resulted  in  the  longest 
search  times,  on  average  10.26  seconds,  which  was  1.5  seconds  longer  than  the  visual  only 
condition  and  2.8  seconds  longer  than  the  combined  cue.  These  results  are  consistent  with 
the  findings  of  Begault  et  al.12  and  Simpson  et  al.13  where  a  combined  audio/visual  system 
performed  better  than  either  audio  or  visual  cueing  alone  for  the  TCAS  and  TAS.  Humans 
are  known  to  process  audio  information  faster  than  visual.  However,  during  a  high  workload 
task  where  spatial  information  needs  to  be  derived  from  the  audio  signal,  audio  information 
will  not  always  be  processed  faster  than  visual.  Increase  in  cognitive  demand  may  be  one 
reason  that  the  audio  cue  produced  slower  results  than  the  visual.  Another  cause  for  the 
same  trend  may  be  the  limitation  of  using  non-individualized  HRTFs,  which  are  known  to 
degrade  elevation  resolution.8  The  majority  of  the  subjects  commented  on  the  fact  that  they 
had  difficulty  interpreting  the  audio  cue  when  the  targets  were  located  at  ±40°  elevation  for 
any  azimuth,  with  the  greatest  difficulty  occurring  at  0°  azimuth. 
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CONCLUSIONS 


The  NDFR  display  has  been  shown  to  be  as  effective  as  the  current  Mil-Std  display  in 
providing  attitude  information.  The  cleaner,  more  integrated  display  of  the  NDFR  provided 
subjects  with  all  the  needed  attitude  information  while  reducing  their  visual  search  time. 
Though  more  training  may  be  required  to  become  accustomed  to  the  NDFR  display,  its 
ability  to  provide  attitude  information  quickly  to  the  pilot  when  looking  on  or  off-boresight 
may  help  reduce  spatial  disorientation.  This  experiment  required  the  subjects  to  fly  a  simple 
straight  and  level  flight;  future  work  should  look  into  the  benefits  of  the  NDFR  display 
during  a  more  dynamic  flight  profile. 

Using  the  Non-Distributed  Flight  Reference  display  for  attitude  information  combined  with 
a  3-D  audio/ visual  cueing  system  for  target  acquisition  is  a  promising  method  of  reducing 
search  time  during  a  visual  search  task.  Although  this  experiment  did  show  that  a  combined 
audio/ visual  cueing  system  was  faster  than  either  alone,  future  experiments  should 
incorporate  all  of  the  lessons  learned  from  past  localized  audio  research.  In  this  experiment 
the  voiced  cue  did  not  contain  elevation  information  that  has  proven  to  be  effective  in  past 
experiments.13  As  the  Air  Force  moves  forward  in  improving  visual  systems,  including 
localized  audio  cues,  and  developing  tactile  sensors,  more  research  should  be  conducted  that 
incorporates  these  three  media. 
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APPENDIX  A 

Consent  Form 


CONSENT  TO  PARTICIPATE  IN 
NON-BIOMEDICAL  RESEARCH 

COMBINED  EFFECTS  OF  A  3-D  AUDITORY/VISUAL  CUEING  SYSTEM  AND 
THE  N ON-DISTRIBUTED  FLIGHT  REFERENCE  ON  VISUAL  TARGET 
DETECTION  USING  A  HELMET-MOUNTED  DISPLAY 

You  are  asked  to  participate  in  a  research  study  conducted  by  L.  Young,  Sc.D.  and  C. 
Pinedo,  from  the  Aero/Astro  Department  at  the  Massachusetts  Institute  of  Technology 
(M.I.T.)  The  results  of  the  study  will  contribute  to  thesis  work  done  by  Carlos  Pinedo.  You 
were  selected  as  a  possible  participant  in  this  study  because  you  are  18  years  of  age  or  older 
and  have  no  serious  sight  or  hearing  impairments.  You  should  read  the  information  below, 
and  ask  questions  about  anything  you  do  not  understand,  before  deciding  whether  or  not  to 
participate. 


.  PARTICIPATION  AND  WITHDRAWAL 

Your  participation  in  this  study  is  completely  voluntary  and  you  are  free  to  choose  whether 
to  be  in  it  or  not.  If  you  choose  to  be  in  this  study,  you  may  subsequently  withdraw  from  it 
at  any  time  without  penalty  or  consequences  of  any  kind.  The  investigator  may  withdraw 
you  from  this  research  if  circumstances  arise  which  warrant  doing  so. 


.  PURPOSE  OF  THE  STUDY 

It  is  the  objective  of  this  study  to  investigate  the  combined  benefits  of  the  Non  Distributed 
Flight  Reference  (NDFR)  symbology  developed  by  the  Air  Force  and  a  3-D  audio/ visual 
cueing  system  for  target  acquisition  tasks  in  a  stationary  simulator  using  a  helmet  mounted 
display  (HMD). 

.  PROCEDURES 

If  you  volunteer  to  participate  in  this  study,  we  would  ask  you  to  do  the  following  things: 

The  study  will  begin  with  a  quick  Powerpoint  presentation  that  describes  in  greater 
detail  the  attitude  displays  and  cueing  system.  After  finishing  the  presentation  you  will  be 
asked  to  fly  two  practice  runs  so  as  to  become  familiar  with  the  two  attitude  displays.  Once 
you  feel  comfortable  with  the  system  the  experiment  will  begin. 
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You  are  asked  to  fly  two  trials  where  you  will  visually  acquire  360  targets.  The 
simulation  will  be  displayed  using  a  helmet  mounted  display  system.  The  audio  cues  will  be 
given  through  headphones. 

During  each  trial  you  are  to  fly  through  a  series  of  waypoints.  Optimum  performance 
is  defined  as  flying  straight  from  the  center  of  one  waypoint  to  the  center  of  the  next 
waypoint.  During  the  flight  you  will  be  asked  to  locate  and  indicate  a  series  of  aerial  targets. 
You  will  be  notified  that  you  have  acquired  a  target  and  can  begin  locating  the  next  target. 
Your  flight  performance,  as  well  as  your  time  to  acquire  a  target  will  be  measured.  Head 
movement  and  velocity  will  be  measured  to  study  search  strategies. 

The  two  trials  are  scheduled  to  take  no  longer  than  thirty  minutes  each.  After 
completing  the  trials  you  will  be  asked  to  fill  out  a  subjective  questionnaire  to  help  us  better 
understand  how  adequate  the  systems  are.  The  entire  study  should  take  no  longer  than  1.5 
hours. 

.  POTENTIAL  RISKS  AND  DISCOMFORTS 

Due  to  the  nature  of  the  experiment  you  may  begin  to  feel  motion  sick.  If  flying  the 
simulator  is  making  you  feel  motion  sick  please  respond  verbally  that  you  would  like  to  stop 
the  simulation. 

.  POTENTIAL  BENEFITS 

The  study  posses  no  potential  benefits  to  you  as  a  subject,  however,  your  participation  in  this 
study  is  greatly  appreciated.  This  research  may  potentially  help  combat  spatial  disorientation 
in  high  performance  aircraft  as  well  as  help  design  systems  that  increase  the  performance  of 
pilots  in  fighter  aircraft. 


.  PAYMENT  FOR  PARTICIPATION 

For  participation  in  this  study  you  will  receive  a  payment  of  $1 0.00/hr. 

.  CONFIDENTIALITY 

Any  information  that  is  obtained  in  connection  with  this  study  and  that  can  be  identified 
with  you  will  remain  confidential  and  will  be  disclosed  only  with  your  permission  or  as 
required  by  law. 

No  personal  information  will  be  kept.  You  will  be  assigned  a  number  so  as  to  keep  track  of 
the  data.  The  data  itself  will  be  disclosed  only  to  members  of  the  Man  Vehicle  Lab  and  the 
Air  Force  Research  Lab  located  at  Wright  Patterson  AFB. 

All  electronic  data  will  be  kept  on  a  password  protected  computer.  All  other  information  will 
be  kept  in  the  experimental  room,  which  is  always  locked. 
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IDENTIFICATION  OF  INVESTIGATORS 


If  you  have  any  questions  or  concerns  about  the  research,  please  feel  free  to  contact  Carlos 
Pinedo,  astech@mit.edu. 


.  EMERGENCY  CARE  AND  COMPENSATION  FOR  INJURY 

In  the  unlikely  event  of  physical  injury  resulting  from  participation  in  this  research  you  may 
receive  medical  treatment  from  the  M.I.T.  Medical  Department,  including  emergency 
treatment  and  follow-up  care  as  needed.  Your  insurance  carrier  may  be  billed  for  the  cost  of 
such  treatment.  M.I.T.  does  not  provide  any  other  form  of  compensation  for  injury. 
Moreover,  in  either  providing  or  making  such  medical  care  available  it  does  not  imply  the 
injury  is  the  fault  of  the  investigator.  Further  information  may  be  obtained  by  calling  the 
MIT  Insurance  and  Legal  Affairs  Office  at  1-617-253  2822. 

.  RIGHTS  OF  RESEARCH  SUBJECTS 

You  are  not  waiving  any  legal  claims,  rights  or  remedies  because  of  your  participation  in  this 
research  study.  If  you  feel  you  have  been  treated  unfairly,  or  you  have  questions  regarding 
your  rights  as  a  research  subject,  you  may  contact  the  Chairman  of  the  Committee  on  the 
Use  of  Humans  as  Experimental  Subjects,  M.I.T.,  Room  E32-335,  77  Massachusetts  Ave, 
Cambridge,  MA  02139,  phone  1-617-253  6787. 
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SIGNATURE  OF  RESEARCH  SUBJECT  OR  LEGAL 
REPRESENTATIVE 


I  understand  the  procedures  described  above.  My  questions  have  been  answered  to  my 
satisfaction,  and  I  agree  to  participate  in  this  study.  I  have  been  given  a  copy  of  this  form. 


Name  of  Subject 


Name  of  Legal  Representative  (if  applicable) 


Signature  of  Subject  or  Legal  Representative  Date 


SIGNATURE  OF  INVESTIGATOR 


In  my  judgment  the  subject  is  voluntarily  and  knowingly  giving  informed  consent  and 
possesses  the  legal  capacity  to  give  informed  consent  to  participate  in  this  research  study. 


Signature  of  Investigator 


Date 
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APPENDIX  B 

Subjective  Questionnaire 


Pilot 


COMBINED  EFFECTS  OF  A  3-D  AUDITORY/VISUAL  CUEING  SYSTEM  AND 
THE  N ON -DISTRIBUTED  FLIGHT  REFERENCE  ON  VISUAL  TARGET 
DETECTION  USING  A  HELMET-MOUNTED  DISPLAY 


Thank  you  again  for  participating  in  this  study  please  fill  out  the  information  below: 


Age 


How  often  do  you  play  video  games? 

□  Never 
EH  Daily 

□  Weekly 
EH  Monthly 

Gender 

□  Male 

□  Female 

Dominant  Hand: 

EH  Right  Hand 
EH  Left  Hand 

Flying  Experience 

□  No  previous  flying  exerperience 

□  Prior  Flying  Experience 

□  Aircraft 

#  of  hours _ 

EH  Helicopter 

#  of  hours _ 


Do  you  use  any  corrective  eye  wear? 

EH  None 
EH  Eye  gl  asses 

□  Contacts 

Are  you  wearing  any  corrective  eye  wear 
now? 

EH  Eye  gl  asses 

□  Contacts 

Have  you  ever  suffered  from  hearing  loss 
or  have  any  hearing  impairments? 

EH  Yes 

□  No 
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Figure  21:  Subject  Altitude  RMSE 
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